Modular Neural Networks and Self-Decomposition

To embed modularity (i.e. to perform a local and encapsulated computation) into neural networks (NN) leads to many advantages. Hence, the development of a general model of modular neural networks (MNN) will enable a broader use of Neural Networks (NN). However, some important issues remain to be solved to enable a systematic use of MNN. In a practical point of view, the most important matter concerns the decomposition of the task into subtasks. We have introduced here the concept of vertical and horizontal decomposition in order to classify the existing modular models capable of performing a self-decomposition. The modular models available for a horizontal self-decomposition (i.e. a clustering of the input space) are mainly the Local Model Network (LMN) and the algorithm of Jacobs and Jordan. Those two algorithms appear complementary. The convergence of the latter one is not ensured but the criterion it uses for decomposing the input space is far more ambitious and eecient than the spatial one used by the LMN. However, the convergence of the LMN is almost ensured. Moreover, its learning and decomposition capabilities can be increased by using polynomial local models. Then, from the complementarity of both those algorithms it would be very eecient to develop a new model performing a horizontal self-decomposition. A review of the algorithms performing a vertical decomposition (i.e. the decomposition of the input variables into packs involving each diierent sub-function) shows clearly that there is no algorithms performing a vertical self-decomposition for the moment. Because this method involves some very interesting features like decomplexiication of the task and meaningful and clear 1 neural representation, it appears important to investigate deeply this topic in an interdisciplinary framework.